feat(orchestrate): per-role model selection — cheap worker, strong planner/auditor - #63
Merged
Merged
Conversation
…anner/auditor dev-loop launched every worker as a bare `claude --permission-mode <mode>`, so the implementer, the planner, and the test-quality auditor all ran on whatever single model the user had configured. That couples three jobs with opposite cost/quality profiles: the implementer is the token-heavy one and the plan and the audit are where a weak model does the most damage. Split them along the seams that already existed: - Worker model is now settable via `--model` / `DEV_LOOP_WORKER_MODEL` in orca-worker-start.sh, orca-spawn.sh, and launch-session.sh. Unset omits the flag entirely, so existing deployments are byte-identical on upgrade. The value is whitelisted (alphanumerics, . _ - and the [1m] suffix) and single-quoted, matching how `--perm` and `--agent` are already guarded. Inert on the non-escalation path, where Orca composes the agent command itself — documented in the script header alongside the same caveat for `--permission-mode`. - test-quality-auditor is pinned (`model: opus`) instead of `inherit`. As the self-grading guard it must not follow the worker down: an inheriting auditor grades a cheap worker at the worker's own tier, so writer and grader share blind spots — the exact failure the agent exists to prevent. - Planning moves to the coordinator. wiki-plan now runs in the coordinator session (Phase 3 step 2a) and the worker ADOPTS the resulting plan via loop-implement's existing "a plan already exists" entry mode, reporting a gap rather than re-planning. The phase sequence is untouched: the worker still signals plan_ready, so watch-status, the ready-set scheduler, and the loop-gate allowlist all see what they saw before. A plan is where an unmade decision becomes the implementer's guess, so it must come from the strongest model in the run rather than the tier that happens to be executing. Tests: 436 pass, 0 fail. New coverage on all three launch scripts for the normal, error (shell-metacharacter injection rejected, nothing created), and boundary (unset adds no flag; `opus[1m]` accepted) cases. The Orca prompt-set checksum in send-prompt.bats is bumped in this commit, as that test requires. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…del lever Claude Fable 5 access is confirmed on this account, so the self-grading guard runs on the strongest available tier instead of Opus. Verified rather than assumed: `--output-format json` reported modelUsage.canonicalModel "claude-fable-5", provider firstParty, 1M context — not a silent fallback. Also spells out what step 2a implies but never said: because planning now runs in the coordinator session, the planning model IS the coordinator's model. There is no third setting — start the coordinator with `claude --model <tier>` and leave DEV_LOOP_WORKER_MODEL on the cheaper implementer tier. Test-isolation fix in the same pass: the three "unset adds no --model" boundary tests read the ambient environment, so once DEV_LOOP_WORKER_MODEL was set as a real user setting they failed — they were asserting the developer's shell, not the code. They now pass `env -u DEV_LOOP_WORKER_MODEL` explicitly. Suite is green (436/0) *with* the variable exported, which is what makes the fix real. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
choiyounggi
added a commit
that referenced
this pull request
Aug 7, 2026
Bumps .claude-plugin/plugin.json (checked against the pushed v* tag by release.yml) and .claude-plugin/marketplace.json (marketplace listing, unenforced) together, so the unenforced one cannot drift. Ships #63: worker model via --model / DEV_LOOP_WORKER_MODEL, the auditor pinned off `inherit`, and wiki-plan moved to the coordinator so the planning tier is independent of the implementing tier. Minor rather than patch: new configuration surface, backward compatible — unset changes nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
dev-loop launched every worker as a bare
claude --permission-mode <mode>, with no--model. So the implementer, the planner, and the test-quality auditor all ran on whichever single model the user had configured.Those three jobs have opposite cost/quality profiles. The implementer is the token-heavy one; the plan and the audit are where a weak model does the most damage. Coupling them means you either overpay for implementation or under-power the two judgment steps — there was no way to choose.
What changed
1. Worker model is selectable —
--model/DEV_LOOP_WORKER_MODELonorca-worker-start.sh,orca-spawn.sh,launch-session.sh.._-, and the[1m]suffix) and single-quoted — the same guard--permand--agentalready get.--permission-mode.2.
test-quality-auditoris pinned (model: opus) instead ofinherit.It is the self-grading guard, so it must not follow the worker down: an inheriting auditor grades a cheap worker at the worker's own tier, making writer and grader share blind spots — the exact failure this agent exists to prevent. Raise the pin, never lower it.
3. Planning moves to the coordinator.
wiki-plannow runs in the coordinator session (Phase 3 step 2a), and the worker adopts the resulting plan through loop-implement's already-existing "a plan already exists" entry mode — reporting a gap rather than silently re-planning (re-planning would move the decisions back onto the worker tier).The phase sequence is deliberately untouched: the worker still signals
plan_ready, sowatch-status, the ready-set scheduler, and the loop-gate allowlist all see exactly what they saw before. A plan is where an unmade decision becomes the implementer's guess, so it should come from the strongest model in the run — not from whatever tier happens to be executing.Tests
436 pass, 0 fail (full suite, macOS).
New coverage on all three launch scripts:
claudecommand--model;opus[1m]is acceptedorca-worker-start.batsadditionally covers env-default, flag-overrides-env, and precedence.The Orca prompt-set checksum in
send-prompt.batsis bumped in the same commit as the O1 prompt edit, as that test's own comment requires.Notes for review
opus. If this account has Claude Fable access,model: fableis a one-word change — it was left atopusbecause Fable availability could not be verified from here.--fallback-modelis not usable for workers: it only works with--print, and workers are interactive tmux/Orca sessions. So the model has to be chosen up front rather than falling back at runtime.🤖 Generated with Claude Code